• Static word embeddings
  • Frequency based methods, word2vec, GloVe, fastText, evaluation of embeddings
• Contextual word embeddings
  • ELMo, Transformers and attention, BERT, sentence embeddings, contrastive learning
• Additional topics
  • Geometry of the embedding space, bias, sentiment, multilingual embeddings
• Topological data analysis
  • Hyperbolic embeddings, singularities and topological polysemy

• The trophy doesn’t fit into the brown suitcase because it’s too large.
• The trophy doesn’t fit into the brown suitcase because it’s too small.

Task: Co-reference resolution

• The city councilmen refused the demonstrators a permit because they feared violence.
• The city councilmen refused the demonstrators a permit because they advocated violence.

Task: Co-reference resolution

• easy for humans to solve
• difficult for computers
  • solution relies on real-world knowledge and common sense reasoning

• I put the cake away in the refrigerator. It has a lot of butter in it.
• I put the cake away in the refrigerator. It has a lot of leftovers in it.

• The old man the boat.

• The complex houses married and single soldiers and their families.

• The horse raced past the barn fell.

“You shall know a word by the company it keeps.” (J. R. Firth, 1957)

(Jurafsky and Martin 2009) (https://web.stanford.edu/~jurafsky/slp3/)

Some of the word vectors from a 100 dimensional fastText embedding trained on a Wikipedia corpus; projected to 2 dimensions using t-SNE.

• Word-sense induction (WSI) or word-sense discrimination: task is the identification of the senses/meanings of a word
• Output: clustering of contexts of the target word, or a clustering of words related to the target word

Example:

• target word “cold”
• collection of sentences:
  • “I caught a cold.”
  • “The weather is cold.”
  • “The ice cream is cold.”

Output: ?

• Word-sense disambiguation (WSD): relies on a predefined sense inventory, and the task is to solve the ambiguity in the context
• Output: identifying which sense of a word is used in a sentence

• grammatical tagging: decide which part of speech (noun, verb, article, adjective, preposition, pronoun, adverb, conjunction, and interjection) a word in a text corpus belongs to

PoS might depend both on definition of the word and its context

• in language a large portion of word-forms are ambiguous
• example from Wikipedia:
  • “dogs” usually is a plural noun,
  • but can also be a verb as in the sentence “The sailor dogs the hatch.”
• example where order matters:
  • “can of fish”
  • “we can fish”

Sub-categories for PoS tagging:

• for nouns, the plural, possessive, and singular forms can be distinguished.
• “case” (role as subject, object, etc.), grammatical gender, and so on
• verbs are marked for tense, aspect, and other things

Other tagging tasks:

• Document classification: spam / not spam

• Review classification: positive / negative

• Sentiment: positive / neutral / negative

• single-label classification / multi-label classification

(https://lena-voita.github.io/nlp_course.html)

• Generative models:
  • learn undelying data distribution \(P(x, y) = P(x | y) \cdot P(y)\)
  • prediction: given an input \(x\), pick a class with the highest joint probability \(y = \mathop{\mathrm{argmax}}_{k} P(x | y = k) \cdot P(y = k)\)
    • maximum a posteriori (MAP) estimate
• Discriminative models:
  • learn the boundaries between classes (i.e. learn how to use the features)
  • prediction: given an input \(x\), pick a class with the highest conditions probability \(y = \mathop{\mathrm{argmax}}_{k} P(y = k | y)\)
    • maximum likelihood estimate (MLE) of parameters

Bag of Words (BoW) assumption: word order does not matter

term-document matrix:

• each document is represented by a vector of word counts

term frequency – inverse document frequency (tf-idf):

• sparse vectors
• words are represented as a simple function of the counts of neighbors

word2vec (Mikolov, Chen, et al. 2013; Mikolov, Sutskever, et al. 2013)

(Jurafsky and Martin 2009)

GloVe (Pennington, Socher, and Manning 2014)

(https://nlp.stanford.edu/projects/glove/)

fastText (Bojanowski et al. 2016)

• I’m going to the bank to withdraw some money.
• We’re sitting on the river bank with some friends.

(https://huggingface.co/tasks/sentence-similarity)

• train on sentence-pair regression tasks like semantic textual similarity (STS)

Sentence-BERT (Reimers and Gurevych 2019)

(https://www.sbert.net/)

(Nickel and Kiela 2017)

• Poincaré GloVe (Tifrea, Bécigneul, and Ganea 2018)

Hyperbolic image embeddings (Khrulkov et al. 2019)

• manifold hypothesis does not hold at all points of certain static word embeddings

(Jakubowski, Gasic, and Zibrowius 2020)

• topological polysemy: count the number of “meanings” around a singularity

• Schedule:
  • Google sheet with dates
  • Detailed program
• Gebäude 25.12 / 2512.02.33 (& live stream available online)
• Each week talks by students (1 or 2 speakers per session, 70 minutes in total)
  • there should be enough time for questions and a discussion
• Guest lecture “Multilingual embeddings” on 2022-06-30
• The final grade is based on your presentation
• Hand in your extended abstract (ideally .tex, .bib files and compiled .pdf; maximum 2 pages with references) via ILIAS
  • ACL template on Overleaf